Enhanced descriptive captioning model for histopathological patches

Abstract The interpretation of medical images into a natural language is developing field artificial intelligence (AI) called image captioning. This integrates two branches which are computer vision and processing. challenging topic that goes beyond object recognition, segmentation, classification since it demands an understanding the relationships between various components in how these objects function as visual representations. content-based retrieval (CBIR) uses captioning model to generate captions for user query image. common architecture systems consists mainly feature extractor subsystem followed by caption generation lingual subsystem. We aim this paper build optimized histopathological stomach adenocarcinoma endoscopic biopsy specimens. For extraction subsystem, we did evaluations; first, tested 5 different models (VGG, ResNet, PVT, SWIN-Large, ConvNEXT-Large) using (LSTM, RNN, bidirectional-RNN) then compare with (LSTM-without augmentation, LSTM-with augmentation BioLinkBERT-Large embedding layer-with augmentation) find accurate one. Second, 3 concatenations pairs (SWIN-Large, PVT_v2_b5, get among them most expressive extracted vector pre-trained compared LSTM both evaluations, select from model. Our experiments showed building system concatenation ConvNEXT-Large PVT_v2_b5 extractor, combined produces best results other combinations.